(19) 



J 



Europii8^e/^*^itentamt 
European Paw<it Office 
Office europ6en dee brevets 




(12) 



(43) Date of publication: 

29.01.2003 Bulletin 2003/05 

(21) Application number: 02014717.9 

(22) Date of filing: 03.07.2002 



(11) EP 1 280 068 A2 

EUROPEAN PATENT APPLICATION 

(51) lnlCl7: G08F 17/27 



(84) Designated Contracting States: 


• Wang, Zl/I 


AT BE BG CH CY CZ DE DK EE ES Fl FR GB GR 


Redmond, Washington 98052 (US) 


IE IT U LU MC NL FT SE SK TR 


• Wolff, Roger 


Designated Extension States: 


Redmondp Washington 98052 (US) 


ALLTLVMKRO SI 


• Huynh, Tuan 




Seattle, Washington 98121 (US) 


(30) Priority: 16.07.2001 US 906467 


• Higashiyama, Nobuya 




fssaquah, Washington 98029 (US) 


(71) Applicant: MICROSOFT CORFORATION 


• Ammeriaanp Michael 


Redmond, WA 98052 (US) 


Sammamlsh, Washington 98053 (US) 


(72) inventors: 


(74) Representative: Qrunedter, KInkeldey, 


• Reynar, Jeff 


Stodonalr & Schwanhausser Anwaltssoztetat 


Woodinvllle, Washington 98072 (US) 


Maximillanstrasse 58 




80538 MOnchen (DE) 



(54) AppJicatlon program interfaces for seniantioaliy labeling strings and providing actions 
based on semantically labeled strings 



(67) Application program Interfaces (API) are pro- 
vided for labeling strings while a user is creating a doc- 
ument and providing user actions based on the type of 
semantic label applied to the string. A recognizer API is 
provided and includes properties and methods or in- 
structions which allow recognizer plug-ins to semanti- 
cally label strings of text or cells or information. An action 
API is provided and Includes properties and methods 
that are called upon when a user initiates particular ac- 
tions such as opening a web browser, going to a partic- 
ular URU or opening an Instance of a word processing 
or spreadsheet program. After the strings are annotated 
with a type label, application program modules may use 



the type label to provide users with a choice of actions. 
If the user's computer does not have any actions asso- 
ciated with a type label, the user may be provided with 
the option to surf to a download Uniform Resource Lo- 
cator (URL) and download action plug-ins for that type 
label. One or more recognizer plug-ins perfomi the rec- 
ognition of particular strings in an electronic document. 
The recognizer plug-ins may be packaged with an ap- 
plication program module or they may be written by third 
parUes to recognize particular strings that are of interest. 
One or more action plug- Ins provide possible actions to 
be presented to the user t>ased upon the type label as- 
sociated with the string. 
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Description 
T9chnicfll Field 

5 (0001 ] This Invention relates to application program interfaces for semantically labeling strings of text during creation 
of an electronic document artd providing a selection of actions that may be performed based on the semantically labeled 
strings. 

Background of the Invention 

10 

[0002] Electronic documents such as word processing documents and spreadsheet documents typlcatly include 
semantic Infonmation that would be helpful if the information was recognized as such. Recognition and use of this 
semantic infonmation could result in Increased interoperability between desktop software applications and other desktop 
applications and/or web-based applications. 
15 [0003] . The ability to recognize strings of text, such as In search engines, is well-known. Additionally, various Infor- 
matk)n retrieval systems have capabilities to label documents. For example, the LEXIS-NEXIS service provides links 
in some of its documents for company names, famous people and stock ticker symbols. 

[0004] However, none of the services described above allow strings of text to be labeled with semantic information 
on-the-fty, I.e., as a user is typing text Into a document and creating a docunDent. Thus, there Is a need for a method 
^ and system for semantically labeling strings while a user is creating a document and providing user actions based on 
the type of semantk: labet applied to the string. There is further a need for application program interfaces (API) for 
lai^eNng strings while a user is creating a document and providing user acttons based on the type of semantic label 
applied to the string. 

[0005] It is with respect to these considerations and others that the present Invention has been made. 

25 

Summary of the Invention 

[0006] The present invention provides application program interfaces (API) for labeling strings while a user is creating 
a document and providing user actions based on the type of semantic label applied to the string. A recognizer API is 

30 provided and Includes properties and methods or instructions which allow recognizer plug-ins of a recognizer dynamic- 
link library (DLL) to semantically label strings of text or cells or information. An action API is provided and inckjdes 
properties and methods that are called upon when a user initiates particular actions such as opening a web browser, 
going to a particular URL, or opening an Instance of a word processing or spreadsheet program. 
[0007] These and other features, advantages, and aspects of the present invention may be more cieariy understood 

35 and appreciated from a review of the following detailed description of the disclosed embodiments and by reference to 
the appended drawings and claims. 

Brief Description of the Drawings 

40 [0008] 

Fig. 1 is a block diagram of a computer that provides the exemplary operating environment for the present Invention. 
Fig. 2 is a block diagram illustrating an exemplary architecture for use in conjunction with an embodiment of the 
present invention. 

45 Fig. 3 is a flow chart iitustrating a method for semantically labeling strings during creation of an electronic document. 

Rg. 4 is an illustratton of a display of a semantic category and its associated dropdown menu. 
Fig. 5 is a block diagram Illustrating properties and methods associated with an action API and a recognizer API. 

Detailed Description 

50 

[0009] The present invention is directed toward application program Interiaces (API) for labeling strings while a user 
is creating a document and providing user actions based on the type of semantic label applied to the string. A recognizer 
API is provided and includes properties and methods or instructions whk:h allow recognizer plug-ins of a recognizer 
dynamic-link library (DLL) to semantically label strings of text or cells or Information. An action API is provided and 
55 Includes properties and methods that are called upon when a user Initiates particular actions such as opening a web 
browser, going to a particular URL. or opening an instance of a word processing or spreadsheet program. A siring is 
defined as a data structure composed of a sequence of characters usually representing human-readable text. 
[0010] After the strings are annotated with a type label, application program modules may use tlie type label to 
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provide users with a choice of actions. If the user's computer does not have any actions associated with a type label, 
the user may be provided with the option to surf to a download Unifomi Resource Locator (URL) and download action 
plug-ins for that type label. One or more recognizer plug-Ins perform the recognition of particular strings In an eiectronic 
document. The recognizer plug-ins may be padcaged with an application program module or they may be written by 
5 tNrd parties to recognize particular strings that are of interest. One or more action plug-ins provide possible actions to 
be presented to the user based upon Ihe type label associated with the string. 

[001 1 ) In one embodiment, the invention is Incoiporated Into a suite of application programs referred to as "OFFICE', 
and more p£irticularly Is incorporated into a preferred word processing application program entitled 'WORD 10.0', a 
preferred spreadsheet application program entitled 'EXCEL 10.0\ a preferred e-mail application program entitled 
w "OUTLOOK 10.0* and a prefenred web browser application program entitled MNTERNET EXPLORER 6", all marketed 
by Microsoft Corporation of Redmond. Washington. Briefly described, the preferred applrcat'on programs allow a user 
to create and edit electronic documents by entering characters, symbols, graphical objects, and commands. 
[0012] Strings are recognized and annotated, or labeled, with a type label. NXex the strings are annotated with a type 
label, application program modules may use the type label and other metadata to provide users with a choice of elec- 
ts tronio commerce actions. If the user's computer does not have any actions associated with that type label, the user 
may be provided with the option to surf to a download Uniform Resource Locator (URL) and download action plug-ins 
for that type label. 

[0013] Having briefly described an embodiment of the present Invention, an exemplary operating environment for 
the present Invention is described bebw. 

20 

Exemplary Operating Environment 

[0014] Referring now to the drawings, in which like numerals represent like elements throughout the several figures, 
aspects of the present invention and the exemplary operating environment wii) be descrlt)ed. 

25 [001 5] Fig. 1 and the following discussion are Intended to provide a brief, general description of a suitable computing 
environment in which the Invention may be Implemented. While the invention will be described m the general context 
of an application program that njns on an operating system in conjunction with a personal computer, those skilled In 
the art will recognize that the invention also may be implemented in combination with other program modules. Generally, 
program modules include routines, programs, components, data structures, etc. that perfomi particular tasks or Imple- 

^ ment partk^ular abstract data types. Moreover, those skilled In the art will appreciate that the invention may be practiced 
with other computer system configuratfons, indudng hand-held devices, multiprocessor systems, microprocessor- 
based or programmable consumer electronics, cell phones, minicomputers, mainframe computers, and the like. The 
invention may also be practk:ed in distributed computing environments where tasks are performed by remote process- 
ing devices that are linked through a communications network. In a distributed computing environment, program mod- 

^ ules may be kxated in boXh local and remote memory storage devices. 

[001 6] With reference to Rg. 1 , an exemplary system for implementing the Invention includes a conventional personal 
computer 20, including a processing unit 21, a system memory 22, and a system bus 23 that couples the system 
memory to the processing unii 21 . The system memory 22 Includes read only memory (ROM) 24 and random access 
memory (RAM) 25. A basic input/output system 26 (BIOS), containing the basic routines that help to transfer inf onmation 

40 between elements within the personal computer 20, such as during start-up, Is stored In ROM 24. The personal com- 
puter 20 further includes a hard disk drive 27, a magnetic disk drive 28. e.g., lo read from or write to a removable disk 
29, and an optical disk drive 30. e.g., for reading a COROM disk 31 or to read from or write to other optical media. 
The hard disk drive 27, magnetic disk drive 28, and optical disk drive 30 are connected to the system bus 23 by a hard 
disk drive Interiace 32, a magnetic disk drive interface 33, and an optical drive interface 34, respectively. The drives 

45 and their associated computer-readable media provide nonvolatile storage for the personal computer 20. Although the 
description of computer-readable media above refers to a hard disk, a removable magnetic disk and a CO-ROM disk, 
it should be appreciated by those skilled in the ari that other types of media which are readable by a computer, such 
as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, and the like, may also be used in 
the exemplary operating environment. 

so [0017] A number of program modules may be stored in the drives and RAM 25, including an operating system 35, 
one or more application programs 36, a word processor program module 37 (or other type of program module), program 
data 38, and other program modules (not shown). 

[0018] A user may enter commands and Infon^atjon into the personal computer 20 through a keyboard 40 and 
pointing device^ such as a mouse 42. Other input devk:e$ (not shown) may include a microphone, joystick, game pad, 
S5 satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 21 through 
a serial port interface 46 that is coupled to the system bus. but may be connected by other Interfaces, such as a game 
port or a universal serial bus (USB). A monitor 47 or other type of display device is also connected to the system bus 
23 via an Interface, such as a video adapter 48. In addition to the monitor, personal computers typically include other 
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peripheral output devices (not shown), such as speakers or printers. 

[001 9] The personal computer 20 may operate in a networked environment using logical connections to one or more 
remote computers, such as a remote computer 49. The remote computer 49 may be a server, a router, a peer device 
or other common netwotic node, and typically includes many or all of the elements described relative to the personal 

9 computer 20, although only a memory storage device 50 has been Illustrated in Figure 1. The logical connections 
depicted in Figure 1 include a local area network (LAN) 51 and a wide area network (WAN) 52. Such networiclng 
environments are commonplace in offices, enterprise-wide computer networks. Intranets and the Internet. 

[0020] When used in a UVN networking environment, the personal computer 20 is connected to the LAN 51 through 
a network interface 53. When used in a WAN networking environment, the personal computer 20 typically includes a 

10 modem 54 or other means for establishing communk^tions over the WAN 52. such as the Internet. The modem 54. 
whk:h may t:>e internal or extemal. is connected to the system bus 23 via the serial port interface 46. In a networked 
environment, program nr^odules depk;ted relative to the personal computer 20, or portions thereof, may be stored in 
the remote memory storage devk^. It will be appreciated that the network connections shown are exemplary and other 
means of establishing a communications link between the computers nr^ay be used. 

'5 [0021] Fig. 2 is a block diagram illustrating an exemplary architecture 200 for use in conjunction with an embodiment 
of ttie present Invention. The architecture Includes an applteation program module 36, such as word processor program 
module 37 (Fig. 1). The applk^tion program module 36 is able to communicate with a recognizer dynsimic-link library 
210 (hereinafter recognizer DLL) and an action dynamic-link library 215 (hereinafter action DLL) as a user is creating 
or editing an electrons document According to a preferred embodiment, the recognizer DU. 210 and the action DLL 

20 215 are combined In a semantic label DLL 205. The recognizer DLL 210 controls a number of recognizer plug-Ins 220. 
The action DLL 215 controls a number of action plug-Ins 225. The action DLL also controls a type-action database 227. 
[0022] In a preferred embodiment, the action plug-ins and recognizer plug-ins are Automation Servers. Automation 
Servers are well-known aoftware components whk^h are assembled into programs or add functk>nalrty to existing pro- 
grams running on the Mterosoft WfNDOWS(S> operating system. Automation Servers may be written In a variety of 

2^ computing languages and may be un-piugged from a program at run time without having to recompile the program. It 
should also be understood that. In a preferred embodiment, the action DLL and recognizer DLL are merged Into a 
single DLL. 

[0023] The recognizer DLL 210 handles the distribution of strings from the electronic document running on the ap- 
plrcation program module 36 to the indlvWual recognizer plug-Ins 220. The recognizer plug-Ins 220 recognize paitfcular 
30 strings In an electronic document, such as a word processing document, a spreadsheet document, a web page, etc. 
The recognizer plug-ins 220 may be packaged with the application program modide 36 or they may be written by third 
parties to recognize particular strings that are of interest. Typically, the recognizer DU. 210 passes strings to the rec- 
ognizer plug-ins 220 In one paragrapli or cell value increments. 

[0024] As part of recognizing certain strings as ir>cluding semantic information, the recognizer plug-ins 220 detemnine 
35 whk^ strings are to be labeled and how they are to be labeled. After receiving these results from the various recognizer 
plug-ins 220, the recognizer DLL 210 sends semantic categories to the applteatlon program module. In a preferred 
embodiment, a semantk; category comprises the recognized string, a type label, and a download URL. A semantic 
category may also comprise metadata. The recognizer plug-ins 220 each run separately and the recognizer DLL 210 
Is responsible for handling the asynchronicity that results from different recognizer plug-Ins retuming results with dif* 
^0 ferent delays. 

[0025] After a string is laiDeled by a recognizer plug-in 220 and a semantte category is sent to the apptk^tlon program 
module 36, the user of the application program module 36 will be able to execute actions that are associated with the 
type label of the semantic category. The action DLL 215 manages the action plug-iris 225 that are nin to execute the 
actions. As with the recognizer plug-ins 220, the action pfug-lns 225 may be packaged with the application program 

^ module 36 or written by third parties to perform particular actions that are of interest to the third party. The action plug- 
Ins provide possible actions to be presented to the user based upon the type label associated with the string. The 
aotk>n DLL 215 determines what type label the semantic category includes and cross-references the type label in the 
type-action database 227 with a list of actions to determine what actions to present to the user. It should be understood 
that« In a preferred embodiment, the type-action database is not used. Instead, the list of actions Is dynamrcally gen- 

so erated for each type by looking In the registry to determine which actions are installed and ttien querying the action 
DLLs to determine which types they apply to. 

[0026] After the user chooses an action, the actbn DLL 2 1 5 manages the appropriate action piug-lns 225 and passes 
the necessary Information between the action plug-Ins and the application program module 36 so that the^ction plug- 
In may execute the desired action. Typically, the application program module sends the action DLL an automation 
55 request to invoke the action the user has selected 

[0027] As described above, the combination of the recognized string, type label, metadata and download URL is 
referred to herein as a semantic category. The type label is a semantic information label. The semantic category may 
also comprise metadata, which are hidden properties of the semantic category. An example of a semantic category 
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may darify the definition. Suppose a user enters the text 'Gone With the Wind" into an electronic document. The string 
*Gone With the Wind' may be Identified as a semantic category of type label 'Book Title' and of type label 'Movie 
Title*. In addition, metadata such as the ISBN number may be returned by the recognizer plug-in to the appflcatron 
program module as part of the semantic category. A download URL may be provided with the type labels "Book Title' 

s and 'Movie Title* In case the user's machine has not stored action plug-fns for these t)^e labels. For example, an action 
for the type label 'Book TIKe' may be "Buy this Book' from an online retailer If the user does not have the action plug- 
in DLL 225 corresponding to 'Buy this book', then the download URL may be used to navigate the user's web browser 
to an appropriate website to download this action plug-in. In other implementations of the rnvention, multiple download 
URI-s may be provided for a single type label. 

to [0028] It should also be understood that the present Invention, In a preferred embodiment, also recognizes 8equer>ces 
of capitalized words that contain function words, and which are likely to be special, but for which there is no type label 
information. These strings are typteaiiy labeled by a grammar checker program module. 

[0029] The actions provided for a semantic category may utilize both the type label and the text of the recognized 
string. For example, a word processor program module may use a grammar checker as a recognizer piug*in to label 
IS strings that are person names. After a string has been labeled as a person's name, the word processor program module 
may, through a standard user interface mechanism, allow users to execute pertinent actions, such as looking up the 
person's name in the contacts folder in a personal Information manager program module, sending electrons mail, or 
searching for the person's rYame In an HR database. 

[0030] Having described an exemplary architecture, an exemplary method 300 for semantically labeling strings dur- 
20 ing document creation will be described t}elow in reference to Figs. 2 and 3. 

Method for Semantically Lat>ellng Strings During Document Creation 

[0031] Fig. 3 is a flow chart illustrating a method 300 for semantically labeling strings during creatbn of an electronk? 

2$ document. Those skilled in the art will appreciate that this is a computer-implemented process that Is carried out by 
the computer in response to Input from the user and instructions provided by a program module. 
[0032] Refening to Fig, 3, the method 300 begins at start step 305 and proceeds to step 310 when a user opens an 
electronic document in appllcatton program noodule 36. In a pretended embodiment, the electronic document is a word 
processing document or a spreadsheet document. However, the invention is not limited to either of these specific types 

00 of electronic documents. 

[0033] At step 310, the appBcatton program module 36 receives a new string, such as when the user enters text» for 
example a new paragraph, into the electronic document or edits a previously entered paragraph. The method 300 then 
proceeds to step 315. 

[0034] At step 315, the paragraph containing the new string is passed from the application program module 36 to 
OS the recognizer DLL 210. The recognizer DLL is responsible for communrcating with the applbation program module, 
managing the Jobs that need to t>e performed by the recognizer plug-Ins, receiving results from the recognizer plug- 
ins and sending semantte category informatten to the application program module. At boot time, the recognizer DLL 
communicates with its recognizer plug-ir>s to determine what languages it supports, what types it can apply, etc. It 
should be understood that, in a pretended embodiment, a paragraph is passed to the recognizer DLL at step 315. 
40 However, in altematrve embodiments, a sentence, the contents of a spreadsheet cell, a section of the document, the 
entire document, etc. may be passed to the recognizer DLL in other words, the present Invention is not limited to 
simply passing a paragraph to the recognizer DLL The method 300 then proceeds to step 320. 
[0035] Still referring lo step 31 5, the application program module 36 typically sends one paragraph at a time to the 
recognizer DLL. In addition, in a preferred embodiment, a grammar checker program module sends all semantic cat- 
egories (without type labels) to the recognizer DLL that have been identified by the grammar checker program module. 
Passing these semantte categories (without type labels) to the recognizer DLL Is important because doing so saves 
each recognizer plug-in from needing to decide whether something is a capitalized string Interspersed with function 
words (a task that would require writing a number of regular expressions: Cap Cap Unc Cap; Cap Unc Cap; etc.). if a 
label is applied by a recognizer plug-in to a string the grammar checker program module labeled, the grammar checker 
so label will then be removed. 

[0036] At step 320, during Idle time, the paragraph (and infonmation from the grammar checker program module) Is 
passed to the recognizer plug-ins. The method then proceeds to step 325. 

[0037] h should be understood that, in a preferred embodiment, the recognizer DLL 210 maintains a job queue. If 
before the recognizer DLL 210 sends the paragraph to the recognizer plug-ins 220 the user edits the paragraph, then 
ss the job containing the edited paragraph is deleted and is not sent to the recognizer p!ug-ins. Then, a new job enters 
the queue at step 315 after the edited paragraph is received at step 310. This job deletion Is necessary to prevent the 
recognizer plug-ins from performing unnecessary work on a paragraph that has been edited. 
[0038] At step 325, the recognizer plug-ins are executed on the text lo search for special strings. For example the 



EP 1 280 068 A2 



recognizer p(ug*ir)8 are executed on the paragraph to recognize keywords or perform other actions defined by the 
recognizer plug-in. As part of executing the recognizer plug-in. the paragraph nr)ay be broken Into sentences by the 
recognizer plug-?n. However, each recognizer plug-in is responsible for Its own sentence-breaking. After the keywords 
are found at step 325. then the method proceeds to step 330. 
s [0039] At step 330. the results from each of the recx>gnlzer plug-Ins are received by the recognizer DLL. The method 
then proceeds to decision step 335. 

10040] At decision step 335. it is determined whether the paragraph that has been reviewed by the recognizer plug- 
ins has been edited after the paragraph was sent to the recognizer DLL. if so, then the method 300 returns to step 315 
and the edited paragraph Is received by the recognizer DLL from the application program module, if not, then the 

fo method proceecls to step 340. 

[0041] At step 340. the results from the recognizer plug-Ins are compiled into semantic categon'es t)y the recognizer 
DLL and the semantic categories are sent to the application program module. At step 345, the application program 
module displays ttie semantk: categories to the user in the electronic document. The method 300 then ends at step 399. 
[0042] As should be understood from the above description, the architecture for recognizing semantk: categories 

IS permits third parties to develop recognizer plug*ins to identify strings of one or more partk^ular types. The recognizer 
plug-ins communtoate with the application program module and receive a string from the application program module. 
The recognizer plug-ins may apply recognition algorithms to the string and communicate the Identity of recognized 
strings back to the application program module. 

[0043] After a string Is labeled with a particular type label, the user will be able to execute action plug-ins that pertain 
20 to that type label. The action ptug-ins preferably are COM objects that are executed via oommunk:atlon between the 
applrcation program module and the action DLL. Parameters necessary to execute the action (the XML of the string 
labeled as being of a particular type, the XML of the string representing the current selection) will be passed from the 
appiication program module to the action DLL and, In tum» passed to the action plug-in. 

2s Actions Assigned to Type Labels 

[0044] An architecture lor identifying arul executing a set of actions associated with a semantic category may also 
t)e provided. This architecture comprises actions that apply to a particular type label (e.g. an action for book titles may 
be 'Buy this book from shop.Microsoft.com") and executing those actions when the user so desires. An action is a 
30 user-Initiated function applied to a typed string. For example, adding a name to the contacts folder is one action possible 
for a type label "Person name". 

[0045] There is power and flextblllty that results from allowing third party vendors, such as IT professionals, to design 
and write recognizer plug-Ins and action plug*ins for deployment within an organization or for deployment on the Worid 
Wide Web. Some example actions that may be executed include: 

35 

Schedule a meeting 
Create task 
Display calendar 
Add to contacts folder 

40 

[0046] Look up in contacts fokier, address book, Windows Address Book (WAB), Gk>bal 

Address List (GAL), etc. 
Insert address into document 
45 Send mail to 

Display EXPEDIA map 
Stock quote lookup 
Send instant message to 

so [0047] Different actions may be assigned to different type labels and these type label-action assignments may be 
stored In the type-action database 227. Table 1 below Illustrates some possible type label-action pairings. 



Table 1 



Type Labels 


Actions 


Person name 


Show contact info Add to contacts E-mail Insert address into document Send instant message to 


Date 


Show calendar for that day New task with that due date Schedule meeting that day 
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Table 1 (continued) 



Type Labels 


Actions 


Place 


Display EXPEDIA map Add to contacts 


Address 


Add to contacts 


Phone number 


Add to contacts 


E-mail 


Add to contacts 


Date 


Schedule a meeting 


Task 


Schedule a task 


Meeting 


Schedule a meeting 



IS [0048] For each type label, the type-action database 227 may store a download URL specified by the creator of (he 
type label that users who do not have action*plug-ins or recognizer plug-ins for that semantic category type can go to 
in order to get action ptug*lns and/or recognizer plug-ins. For example, the download URL for the type label 'Book 
Title" might be microsoft.com/8emanticcategories.asp. Once at that web page, a user may be offered downloads of 
various action ptug-lr^s and recognizer plug-Ins. There may also be an option on the user Interface to navigate to the 

20 download URL so that recipients of documents with semantic categories can easily get the action plug*ins for those 
semantic categories. 

Storing Semantic Categories 

25 [0049] Semantic categories may be stored as part of the electronic document along with other document information 
and may be available when a document is transmitted from one computer to another computer. In a preferred emt>od- 
iment, storing semantic categories in an electronic document Is controlled by an "Embed semantic categories' check- 
t>ox. The checkbox Is on by default. Turning it off will prevent semantk categories in the document from i>eing saved. 
The state of the checkbox is per document. The same checkbox controls saving for both .htm and .doc documents. 

30 ' [0050] Checking a 'Save semantic categories as XML properties* checkbox (off by default) will write out the text of 
alt of the semantic categories in the document and their labels In the header of the HTML file in XML {that Is using the 
same tags as are used inline, but surrounded by <xm1> And </xmt>) for easy Identification and parsing by search 
engines and knowledge management systems. 

[0051 ] Semantic categories may be saved as a unique namespace plus a tag name. A namespace is an XML con- 
OS struct for uniquely identifying a group of XML tags that belong to a k)gk:al category. Thus, every semantic category is 
uniquely kJentified by its nametag (e.g.. 'streetname*) in addition to its namespace (e.g.. *schemasmicro$oft-com: 
outlookxontacf) 

[O052] Although the method 300 described at)ove is one method for Identifying semantic categories, there nr^y be 
other mechanisms for identifying semantto categories. One mechanism Is a grammar checker program module (not 
^0 shown) connected to word processor program module 37. Another mechanism is receh/rng a semantic category from 
another electronic document. For example, when text containing a semantk: category is copied from one eiectrorvc 
document and passed into another electronic document of the word processor program module 37, the infomnatlon 
identifying the semantic category is preserved and copied along with the copied text. 

45 Displaying Semantic categories to the User 

[0053] Refening now to Fig. 4, an illustration of a display of a semantic category 400 and Its associated dropdown 
menu 405 will be described. It should be understood that Rg. 4 is an illustration of a semantic categoiy 400 and 
dropdown menu 405 as displayed to a user by the appficatton program module 36. 

so [0054] The siring 410 associated with semantic category 400 Is the string 'Bob Smith'. As shown in Fig. 4, the string 
41 0 of a semantic category 400 may be identified to the user by brackets 41 5. Of course, many other devices such as 
coloring, underlining, icons, etc. may be used to indicate to the user that a particular string Is a semantic category. 
[0055] In a preferred embodiment, when the user hovers a cursor over the string 410 or places the insertion point 
within string 410. then dropdown menu 405 is displayed to the user. The dropdown menu may display a list of actions 

55 associated with a semantic category. The dropdown menu may appear above and to the left of the semantk: categoiy 
string. 

[0056] Typically, the first line of the dropdown menu indicates which string is the semantic category string (Bob Smith 
in Rg. 4) and what type the semantic category is (Person name in Fig. 4). Listed below the first line are actions 420 
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available for the semantic categoiy type, such as 'Send mall to...", 'Insert Address", and "Display contact Information../. 
[0057] The first Item on the drop down menu below the separator line Is "Check for new actions../ 425. "Check for 
new actions.. / 425 will appear only for semantic categories whose download URL Is available to the application program 
module. If selected, 'Check for new actions../ 425 uses the semantic category download URL to navigate the user's 
s web browser to the homepage for the semantic category type applied to the string. For example, suppose new actions 
have been defined for the semantic category type 'person name*. Jf so, then new actions will be downloaded to the 
user's computer after selecting "Check for new actions...' 425. 'Check for new actions...' 425 will be grayed out if a 
download URL Is unavailable for the semantic category. 

[0058] If selected, the 'Remove this semantic category" item 430 deletes the semantic category label from the string, 
w If selected, the 'Semantic categories" Item 435 navigates the user to the semantic categories tab of the autocon-ed 
dialog. 

[0059] It should be understood that the appMcatk>n program module sends a request to the action DLL to determine 
which actions are shown with each semantic category type. 

15 Actions Performed in Association with Semantic categories 

[0060] There are a number of functions that users perfomn on typed data that preferred word processor program 
module 37 and semantic categories will make easier. The functions fall Into three primary categories: 

^ 1) interacting with personal information manager contacts* tasks, meetings, and mail; 

2) interacting with properties on the Wbrid Wide Web or a corporate intranet; and 

3) interacting with other applications on the client machine. 

[0061] A single string may be associated with multiple semantic categories. Every semantic category has a type 
2S label with one or more action plug-Ins defined for the type label. For example, the 'Address' type label may have the 
'Open in Mappoint'. 'Find with Expedia Maps* and 'Add to my Address Book' actions associated with it and each of 
these actions may have a different action plug-in to execute the action, 

[0062] The actions assigned to type labels also depend on the computer that the application program module is 
running on. Thus, if a computer has three actions registered for the type label 'Address*, then all strings with an "Ad- 
30 dress" type label will be assigned to three actions. However, if one of these semantic categories is sent to a computer 
whtoh has only two actions registered for the 'Address* type label, then the user will only be exposed to two acttons 
for this semantk: category. 

Nesting of Semantic categories 

35 

[0063] In an embodiment of the present invention, semantic categories may bG nested inside each other. For exam- 
ple, the string 'George WasWngton' may Include a semantic category wrth type label 'Person Name* for the span 
•George Washington State' and a semantic category with type label 'State' for the span "Washington'. Moreover, two 
semantic categories may cover exactly the same span. For example, the string 'George Washington' may irKlude a 
^ semantic category with type label "Person Name' and a semantic category with type lai>ei "President". 

[0064] Because the preferred application program module 37 will support labeling a single string with multiple type 
labels (e.g. Bob Smith could be a semantic category labeled as a "Person Name" and labeled as a "Microsoft employ- 
ee"), the preferred application program module 37 will use cascade menus on the dropdown menu if multiple semantic 
category types are assigned. 

45 [0065] For example, tiie cascade menu may Include a list of the type labels included in the recognized string. This 
list may include a type label 'Person Name' and a type label 'Microsoft employee". 

[0066] It should be understood that a cascade menu may be used to allow the user to select v^ich t^e label the 
user is interested In and to further select an action after selecting the type label. 

so lr)-document User Interface to Indicate Semantic categories 

[0067] As described above with reference to Fig. 4, the appllcatton program module may Include the option to display 
an in-document user Interface to indicate the location of semantic categories. This in-document user interface may 
use a colored indication to indtoate the location of a semantic category, such as the brackets 415 in Fig. 4. The in- 
ss document user interface will also be able to show nesting ot semantic categories. For example, if Mrchael Jordan is 
labeled as a semantte category with type label "Person Name', Michael Is a semantic category with type label "First 
Name" and Jordan is a semantic category with type label 'Last Name', the document may look like this with tl>e brackets 
indicating semantic categories: 
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([MichaetKJordan]] 

[0068] Of courde, the in-document user Interface may be any soit of indication. For example, in the "EXCEL' spread- 
sheet application progranr), the interface comprises a triangle in the lower right hand portion of a celt to indicate that 

5 one or more semantic categories are present in the cell. 

[0069] Although the present Invention has been described as Implemented In a word processing program module, 
it should be understood that the present invention may be implemented in other program modules, including, but not 
limited to. HTML authoring programs and programs such as the "POWERPOINT'iB) presentation graphics program and 
the "OFFICE" program module, tx)th mari^eted by f^icrosoft Corporation of Redmond, Washington. 

10 [0070] As described above, the semantic category may also include metadata relumed by the recognizer piug-lns. 
For example, a recognizer plug-in that recognizes the trtles of books may return as metadata an ISBN book number 
when it recognizes the title of a book. The ISBN book number metadata may then be used to provide actions. Metadata 
may also be used to disambiguate for actions and searches. For example, suppose a recognizer DLL is linked to a 
corporate emptoyee database to recognize names. When the recognizer DLL recognizes "Bob Smith", It may store 

15 •employeelD=12345'' as metadata in the background- Then, when an action is fired, the text in question will be known 
to reference Bob Smith, empbyee no. 12345 rather than Bob Smith, employee no. 46678. Also, the metadata may 
allow searches to be performed independent of the actual text In a document. So, a search may be conducted on 
"Robert Smith' by looking for employee 1 2345 In the employee databases and by performing a search on the metadata 
for employee number 12345 to find documents with 'Bob Smith* in them. There are also numerous other functions for 

20 metadata. For instance, DHTML could be inserted so special features of a web browser, such as text color or font size, 
may be used. Additionally, data used by other actions may be iriserted such as someone^s e-malt address that could 
be used by the send-mail-to action, a normalized version of the date coukJ be stored to easily Interact with a personal 
information manager, etc. 

25 Implementation of Exemplary Application Program Interfaces 

[0071] According to an exemplary embodiment, application program interfaces (API) are provided for implementing 
the recognizer plug-ins 220 and the action plug-ins 225. discussed above. Referring to Rgs. 2 and 5, the recognizer 
API 250 includes a group of properties and methods or instructions that adow the recognizer plug-ins 220 to semanticaHy 
30 label strings ot text or ceils of information, as cfiscussed with reference to Figs. 2 and 3. The action API 250 includes 
properties and methods or instructtons that are called upon when a user Initiates a partteular action, for example, 
opening a Web browser, going to a particular URL, or opening an instance of a word processing applk>ation, a spread- 
sheet application, and the (Ike. 

[0072] Referring to Fig. 5, the properties and methods of the recognizer API 250 are Illustrated. The Description 

35 property 251 includes a description of the recognizer plug-Ins 220. The Name property 252 Includes a name for the 
recognizer plug-in as It appears in a user interface such as a tools menu or options contained In a dialog box of an 
exemplary application program. The Prog(programmatlc)ID 253 Includes a unique identifier for the object class of the 
recognizer DLL The Recognize method 254 Includes instaictions and routines for recognizing character strings as 
actionable after having been semanticaily labeled, as described above. The LabelCount property 236 describes the 

40 number of semantk: label types the associated recognizer DLL recognizes, for example, book titles, movie titles, etc. 
The DownloadURL property 256 Is the URL that Is navigated to download addltksnal semantic label types or categories. 
The LabelName property 257 iru^ludes unique identifiers for the semantic label types that the recognizer DLL supports. 
[0073] The Description property 231 of the action API 230 includes a descriptk>n of the semantic label action asso- 
ciated with a given action, for example, opening a Web browser. The InvokeVerb method 232 includes the routines 

45 and kistructions that are executed when an action item is Initiated, as described above. The Name property 233 includes 
a title for the given action. The ProglD property 234 includes a unique identifier of the object class for the particular 
action. The LabelCaptbn property 235 includes the caption that v^il be displayed at the top of a user interface such 
as an actions menu. The LabelCount property 236 includes the number of semantic label types recognized by a cor- 
responding recognizer DLL or corresponding recognizer plug-Ins. The LabelName property 237 identifies the types of 

so semantic label actions, for example, opening a Web browser, navigating to a particular URL, opening an instance of 
an applrcallon program, etc. 

[0074] The VerbCaptronFromID property 238 includes the captions for available actions provided in a user interface, 
such as an actions menu. The VerbCount property 239 includes how many actions are supported for a given semantk: 
label type. The Verb ID property 240 returns a unique identifier within the semantic label for use within a given application 
program. TTiis mechanism is supplied so that the semantic label recognizer DLL and recognizer plug-Ins can mix and 
match semantic label actions for various semantte label types supported by the DLl_s and plug-ins. The VerbName- 
FromlD property 241 returns a name to represent the semantic label action for use internally wilhin a given application 
program. For example, for the semantic label action string 'view company Website", Ihe VeritiNameFromlD property 



EP 1 280 068 A2 



241 may return a name such as "viewCompany Website*. 

[0075] The folk>wii>g Is a discussion ot an exemplary Implementation ot the recognizer API 250 and the action API 
230. Following a discussion of various components of these APIs ts exeniplary code written in Mtcrosott Visual Basic 
6.0 for a recognizer DLL and action DLL for locating a 'Fourth Coffee" flavor In a set of coffee flavors. The code and 
the example are exemplary only and are not limiting of the scxipe of the invention described herein. 

Implementing the Recognizer API 

10076] To Implement the recognizer API 250, the ProgID property 253, the Name property 252, and the Description 
property 251 are populated by specifying the name and by providing a description of the particular recognizer DLL. 
Additionally, a locale identifier is passed to the recognizer DLL, for example^ Spanish, Gennnan, English, etc., to give 
identification of the user interface language in which semantic labeling will occur for the particular recognizer DLL. The 
following is exemplary code written in Microsoft Visual Basic 6.0 for populating the ProgID property 253, the Name 
property 252, and the Description property 251 . 



Private Property Get LabelRecognizer^ProgldQ 
As String 



LabelRecognizer^Progld = 

"CoffceFlavorXabclRecogmzer" 
End Property 



Private Property Get LabelRecognizerJName 
(By Val LocalelD As Long) As String 
JLabelRecognizer_Name « 
"Coffee Flavors" 

End Property 



Private Property Get LabeIRecognizer_Desc 
(ByVal LocalelD As Long) As String 
LabelRecognizer^Desc = 
**Directs users Coffee Flavors" 

End Property 



[0077] Next, the number of semantic label types, the list of semantic label action types, and the additional semantic 
labels download location of the recognizer DLL are provided by populating the LabelCount property 255, the LabefName 
property 257 and the DownloadURL property 256. The number of semantic label types supported by the recognizer 
DLL 210 are populated. For example, if a value of two (2) for the LabelCount property is retumed, there will be two 
subsequent cafis made to the LabelName property with a label ID value of one(1) passed in the first call and a value 
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of two passed in the second call. The download locatfon includes the HTTP address used (download URL) to download 
toots associdted with particular semantic labels. If no download URL is available, the value of the property Is set to 
null. The following is exemplary code for populating the LatielCount property 255, the Lat>etName property 257 and 
the DownloadURL property 256. 



Private Property Get LabelRecognizCT^LabelCountO 
As Long 
I^beIRecognizer_LabeICount = 1 



End Proper^ 



Private Property Get LabeIRecognizer_LabelNaine 
(ByVal LabellD As Long) As String 
IfLabellD^l Then 
LabelRecognizer_LabeIName = 
••schemas-fourth-com/fourthcoffeetfflavor" 
End If 
End Property 

Private Property Get LabelRecognizer^DoMoiloadURL 
(By Val LabellD As Long) As String 
LabelRecognizer^DownloadURL = Null 
End Property 



(0078] A list of items such as text in a word processing application or information contained in a cell of a spreadsheet 
Is provided to the Recognize nnethod 254. The text or information is provided to the recognizer plug-ins 220 of the 
recognizer DLL as a text string arKJ the form of the text, for example, paragraph, cell, etc., Is provided. The following 
is exemplary programming code for building a list of strings associated with, for example, coffee flavors to be recognized 
by the recognizer plug-ins 220 of the recognizer DLL 2 1 0. According to an exemplary embodiment, to avoid rebuilding 
the list of strings every time there is a call to recognize a text item, the list Is built and populated In an an-ay when the 
class of semantic label types is created. 
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Dim garTTcnns(6) As String 
Dim gintNumTemis As Integer 

Private Sub Class.InitializeO 
ganTcmis(l)''"latte'' 
garrTerms(2) = "carmclito" 

garrTenns(3) = "verona" 
gaTrTerms(4) = "Columbia blend" 
garrTerm$(5) = "antigua" 
garrTcmis(6) = "kona" 
gintNumTenns = 6 
End Sub 

[0079] After the list of strings to be recognized is built, as described above, the Recognize method 254 is constructed 
that wiii search for strings in the array created above. The search performs a case-insensitive search through the 
passed-in strings of text for each of the terms suppiied. For the example given above, the strings supplied include a 
variety of coffee flavors. Construction of the Recognize method 254 lr>cludes. passing in the language identifier of the 
text strings to be searched by the recognizer DLL, for example. German, English, etc. Additionally, an object is passed 
in to add additional semantic labels. A Property Bag properly allows for the addition of new attributes/value pairs to be 
added to semantic labels. For example, the name "Bob Smith" may be set up as a name type, but the Property Bag 
property allows for the storage of other values Wks employee ID, telephone number, address by adding new attribute/ 
value pairs to the semantic label type. For example, the attribute "employee ID" and the value M23' may be added to 
the name type for "Bob Smith/ Information is also passed in to define the name space in the document, and to specify 
the start position of the string and the length of the string. The following is exemplary programming code for constructing 
the Recognize method 254. 

Private Sub LabelRecognizer_Recogni2er ^Recognize 
(ByVal Text As String, 
ByVal DataType As LabelLib.IF_TYPE. 
ByVal LocalelD As Long, 
ByVal RecognizerSite As 

LabelLib.LabelRecognizerSite) 
Dim intLoop As Integer 
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Dim intlndex As Integer 
Dim intTennLen As Integer 
Dim stlPropertyBag As Labellib JLabelProperties 
Text = LCase(Striiig?=Text) 
For intLoop «= 1 To gintNumTenns 
intlndex = InStr(Text, ganTerms(intLoop)) 
intTermLen = Len(ganTenns(intLoop)) 
Do While intlndex >0 
Set StlPropertyBag = 

RecognizerSite,GetNewProp^Bag 
RecognizefSite.CommitLabel 
"schemas*fourth-cQni/fourthcofree#fIavor", intlndex, 
intTermLen, stlPropertyBag 
intlndex - InStr(intIndex + intTermLen, 
Text, ganTenn5(intLoop)) 
Loop 
Next intLoop 
End Sub 

implementing the Action API 

{0080] After construction of the Recognize method 264, the action API 230 (s constructed by first populating the 
ProgJD property 234. the Name property 233, and the Description property 231 . Population of those properties Is similar 
to population of similar properties forthe recognizer API 250, described above. The following is exemplary programming 
code for populating the ProgID property 234, the Name property 233, and the Description property 231 of the action 
API 230. 

Private Property Get LabelAction_ProgIdO As String 
LabelAction_ProgId = 

•'CoffeeFlavorXabelAction" 
End Property 
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Private Property Get LabelAction_Name 
(ByVal LocalelD As Long) As String 
LabelAction_Name = "Coffee Flavor actions" 
End Property 



Private Property Get LabelAction^Desc 
(By Val LocalelD As Long) As String 
LabelAction_Desc = 
••Provides actions for certain CofTee Flavors" 

End Property 



[0081] The recognizer DLL 210 Is tnfortned of the number, names, and captions of the semantic label types by 
populating the LabelCount property 236. the LabelName property 237, and the LabelCaption property 235, For each 
tabef type supported by the recognizer plug-ins 220 of the recognizer DLL 210, the recognizer DLL 210 is informed of 
the number of actions. If there are three actions for the label type 'persons' and two actions for the label type 'com- 
panies," then a total of five label types are passed to the recognizer DLL A name for each action class Is provided and 
a caption that will appear in an appropriate user interface such as an action menu Is provided. For example, the caption 
"Coffee flavors at Joe's Coffee IHouse' might be provided for an action that will navigate to the Web page of Joe*s 
Coffee House for a semanticafly labeled coffee flavor. In addition, according to an exemplary embodiment^ the appli- 
cation programs applicable to a given action may be identified so that Inapplicable application programs may be dis- 
abled. For example, if the semantically labeled information Is not applicable for use in a spreadsheet, a spreadsheet 
application may be disabled. The following is exemplary programming code for populating the LabelCount, the Label- 
Name and LabelCaption properties. 



Private Property Get LabelAction^LabelCountQ 
As Long 
LabelAction_LabelCotint - 1 
End Property 
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Private Property Get LabelAction^LabelName 
(By Val LabellD As Long) As String 
If LabellD - 1 Then 
LabelAction_LabeINanie - 
" schemas-fomth-coni/fourthcofr6e#flavor " 
End If 
End Property 

Private Propaty Get LabelAction_LabelCaption 
(ByVal LabellD As Long, 
ByVal LocalelD As Long) 
As String 
LabelAction_LabelCaption = 
. ••Selected Coffee Flavors" 
End Property 

[0082} Next» the recognizer DLL is informed of the number names, and identifiers of supp6ft(§d semantic label actions 
by populating the VerbCouni property 239, the VerbID property 240, the VerbCaptfonFromtD property 238, and the 
VerbNameFromID property 241. Population of these properties Is done on a fabel type by label type basis to tell the 
label actions about verbs that are supported. Generating a unique ID for a particular verb Is performed by the action 
DLL 215 which gives the action DLL more ffexibility. For example, a semantic label action DLL can specify the same 
VerbID value for the same action across varying semantic label types, or the DLL can use the same VerfolD for similar 
variants of an action. Tho following is exemplary programming code for populating the VerbCount, VerbID, VerbCap- 
tionFromlD, and VerbNameFromID properties. 



Private Property Get LabelAction_VabCount 
(ByVal LabelName As String) As Long 
If LabelName - " schemas-fourth*com/fourthcofree#flavor " Then 
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LabelAction_Vert)Count = 6 
End If 
End Property 

Private Property Get LabelAction_VcrbID 

(ByVal LabelName As String, 

ByVal Voblndex As Long) 

As Long 
LabelAction.VeibID « Veitlndex 
End Property 

Private Property Get LabelAction^VerbCaptionFromlD 
(ByVal VablD As Long, 
ByVal ApplicationName As String, 
ByVal LocalelD As Long) 
As String 
Select CaseVerbID 
Case 1 

LabelAction^VcrbCaptionFromlD = 
"latte" 
Case 2 

LabelAction_VerbCaptionFromID = 
"carmelito" 
Case 3 

LabelAction_VcrbCaptionFromID = 
"verona" 
Case 4 

LabelAction^VerbCaptionFromlD = 
"Columbia blend" 
Case 5 

LabelAction^VerbCaptionFromlD = 
"antigua" 
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Case 6 

LabelAction_VcrbCaptionFromID * 
"kona" 
End Select 
End Property 



Private Property Get LabelAction^VerbNameFromlD 
(ByVal VeibID As Long) As String 
Select Case VeibID 
Case 1 

LabelAction^VerbNameFromlD « "latte" 
Case 2 

LabelAction_VerbNameFromID = "carmelito" 
Case 3 

LabelAction_VerbNameFromID = "verona" 
Case 4 

LabelAction^VeibNameFromlD = "Columbia blend'* 
Case 5 

LabelActJon_VeibNameFromID - "antigua" 
Case 6 

LabelAction^VerbNameFromlD = "kona" 
End Select 
End Property 



[0083] After those properties are populated, the InvokeVerb method 232 is constructed for invoking the appropriate 
verb when the user selects an action displayed in the semantic label user interface corresponding to the VerbCaption- 
FromID value. According to the exemplary programming code provided below, six actions are provided which are 
hyperlinks that navigate to six coffee flavor Web sites. Construction of the InvokeVert method includes identifyfng the 
verb, identifying the action, and naming the application program 36 so that different actions may be perfonDed depend- 
ing on the context, for example, word processing text versus cell data In a spreadsheet. 

[0084] According to an exemplary embodiment, a pointer may be given to an application program's 36 object model 
so that the action DLL may use the object model to manipulate text In a document, insert text, manipulate data in a 
spreadsheet, etc. The label text, the label type, and contents of a property bag are provided, and that information may 
be provided according to alternate computing languages, for example XML, so that the information may be manipulated 
using alternate language systems, such as XML parsers. This aspect is useful for manipulating multiple nested semantic 
labels, for example, street, city, state, and zip code. The folk)wing is exemplary programming code for constructing the 
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InvokeVerb method 232 of the action API 230. 



Private Sub LabelAction_InvokeVeib 

(By Val VerbID As Long, 

ByVal ApplicationName As String, 

ByVal Target As Object, 

ByVal Properties As LabelLib.ILabelPropCTties, 

ByVaJ Text As String, 

ByVal Xml As String) 
Dim ielntemetExplorer As Variant 
Set ielntemetExplorer = 

CreateObjectCIntemetExplorer Application*") 
With ielntemetExplorer 

Select Case VerbID 
Case 1 

.Navigate2 "www.latte.com" 
Ca$e2 

,Navigate2 "wwwxannelitoxom" 
Case 3 

.Navigate2 "www.verona,com" 
Case 4 

*Navigate2 "www.columbiablend.com'' 
Cases 

.Navigate2 "www-antigua-com" 
Case 6 

.Navjgate2 "wwwiona-com" 
End Select 
•Visible ^ True 
End With 
End Sub 
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Registration of ApplJcation ProgrammSng Interfaces 

[0085] In order tor the application program module, such as a word processing application or a spreadsheet appli- 
cation, to know that the recognizer and action ptug^ins and OU^ constructed and made operable by the construction 
of the APIs, discussed above* are actually plug-Ins and DLLs for use by the application, the plug-ins and DLLs must 
be registered In the operating system registry of the user's computer 20. Accordingly, information is written into the 
registry of the operating system 35 so that any plug-ins and DLLs referred to by the semantic label properties in that 
portion of the system registry will be booted by the applicable application program utilizing the semantic labeling func- 
tionality when that application programs boots. 

[0086] Although the present Invention has been descrSied above as Implemented in a preferred application program 
module, it will be understood that alternative embodiments will become apparent to those skilled in the art to which the 
present invention pertains without departing from its spirit and scope. Accordingly, the scope of the present invention 
is defined by the appended claims rather than the foregoing description. 

Claims 

1. In a system for semantically labeling a string of text in an electronic document created in an application program 
module, a rr>ethod of implementing an application program Interface for further implementing a recognizer plug- 
in, the method comprising the steps of: 

providing an identification, a title, and a description for the recognizer plug-in; 
providing a number of text label types and a downtoad location for additional label types; 
providing a list of text Items for recognition by the recognizer plug-In; and 

causing the recognizer plug-in to search for text items from the list of text items in a selection of text received 
by the recognizer plug-in. 

2. The method of Claim 1. further comprising the step of registering the recognizer plug-in in an operating system 
registry. 

3. The method of Claim 1, wherein the step of providing an Identification, a title, and a description for a recognizer 
plug-in Includes the step of providing the name of the recognizer plug-in for presentation via a user interface. 

4. The method of Claim 1 , wherein the step of providing a number of text label types includes provicfing a number of 
text label types recognizable by the recognizer plug-in. 

5. The method of Claim 1 whereby the step of providing a download location for additional text label types Includes 
providing a uniform resource locator (URL) for obtaining additional text label types. 

6. In a system for semantically labeling a string of text in an electronic document created in an application program 
module, a method of implementing an application program interface for further implementing an action plug-in, the 
method comprising the steps of: 

providing an identification, a title, and a description for the action plug-In; 

providing to a recognizer DLL a number of text label types, a name for each of the number of text label types, 
and a caption for each of the number of text label types; 

providing the recognizer DLL a number of text label actions, a name for each of the number of text label actions, 
and an identification for each of the number of text label actions; and 
displaying a list of the text label actions upon user initiation. 

7. The method of Claim 6, whereby prior to the step of displaying a list of the text label actions upon user initiation, 
further comprising the step of providing names of each of the number of text label actions for presentation In a 

user interface. 

8- The method of Claim 7, wherein the step of providing to a recognizer DI_L a caption for each of the number of text 
label types, further comprises the step of providing captions for displaying in the user interiace each of the number 
of text label types. 
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9. The method of Claim 6, wherein the step of providing to a recognizer DLL a number of text label types. Includes 
providing a number of text label types for which tlie action plug-in provides actions. 

10. The method of Claim € further comprising the steps of registering the recognizer DLL and the action plug-in in a 
s computer operating system registry. 

11. In a system for semantically labeling a string of text in an electronic document created in an application program 
module, a method of implementing an application program interface for further Impfementing a recognizer plug-in 
and an action plug-In, the method comprising the steps of: 

10 

providing an identification, a trtfe, and a description for the recognizer plug-in; 

providing the recognizer plug-ln number of text label types, a fist of text label action types, and a download 
location for additional label types; 

providing the recognizer a list of text Items for recognition by the recognizer plug-in; 
^5 providing to the recognizer piug-fn a number of text label types, a name for each of the number of text label 

types, and a caption for each of the number of text label types; 

providing the recognizer plug-in a number of text label actions, a name for each of the number of text fabel 
actions, and an identification for each of the number of text label actions; 
providing an identification, a title, and a description for the action plug-In; 
20 causing the recognizer plug-In to search for text items from the list of text items in a selection of text received 

by the recognizer plug-in; and 

displaying a list of the text label actions upon user initiation. 

12. The method of Claim 1 1 , further comprising the step of registering the recognizer plug-in and the action plug-in in 
^ an operating system registry. 

13. The method of Claim 11 , wherein the step of providing an identification, a title, and a description for a recognizer 
plug-in includes the step of providing the name of the recognizer plug-in for presentation via a user Interface. 

30 14. The method of Claim 11 , wherein the step of providing a number of text label types includes providing a number 
of text label types recognizable by the recognizer plug*in. 

15. The method of Claim 11 , wherein the step of providing a list of text label action types includes providing a list of 
text label action types associated with an action plug-In and recognizable by the recognizer plug-in. 

35 

16. The method of Claim 11 whereby the step of providing a download location for additional text label types Includes 
providing a uniform resource locator (URL) for obtaining additional text label types. 

17. In a system for semantically labeling a-string of text in an electronic document created in an application program 
40 module, the system including a recognizer plug-in and an application program interface (API) for allowing the 

recognizer plug-in to semantically label strings of text, comprising: 

a description property including a description of the reoognizer plug-in; 
a name property Including a name of the recognizer piug-in; 
45 a programmatic identifier including a unique identifier for the recognizer plug-in; 

a label count property identifying a number of semantic label types recognizable by the recognizer plug-in; 
a label name property including unique identifiers for the semantic label types recognizable by the recognizer 
plug-in; 

a label download URL property for locating additional semantic label types recognizable by the recognizer 
so plug-in; and 

a recognize method including instructions which when executed by a computer cause the recognizer plug-In 
to recognize character strings semantically labeled as character siring types recognizable by the recognizer 
plug-in. 

ss 13. The application program interface of Claim 17, further comprising an action API, comprising 

a description properly describing a semantic label action associated with an action operable on a semantically 
labeled text string; 

a name property including a title for the semantic label action; 
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a programmatic identifier property including a unique identifier for the semantic \abB\ action; 
a label caption property for displaying via a user interface; 

a label name property identifying a number of types of semantic label actions operable by tlie action plug-in; 
a labef count property identifying the number of semantic label types recognizable by the recognizer pti^nn; 
a verb caption from ID property including captions for the number of semantic label actions provided via a 
user interface; 

a verb count property including a quantity of semantic label actions supported by a semantic label type; 
a vert) ID property for retuming a unique identifier within a semantic label for use within the application 
program module; 

a verb name from 10 property for returning a name to represent a semantic label action for use within the 
application program module; and 

an Invoke verb method including instructions wiiich when executed by a computer initiate the semantic label 
action. 
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